Stacked on top of the schema PR (elastic#268485). Targets that branch in this fork. When the schema PR merges to elastic/main, this PR's base will be flipped to elastic/main and re-opened against elastic/kibana for review.

TL;DR

1. Schema: discovery_labels becomes nested { value, kind } with value as search_as_you_type. Title and type are auto-prepended to discovery_labels at index time by the SML indexer. The dedicated title_autocomplete field and the type.autocomplete subfield are dropped — they're no longer needed.

2. New route: POST /internal/agent_context_layer/sml/_autocomplete — dedicated to the @ menu / typeahead. Separate from the existing POST /sml/_search (full retrieval for the LLM tool). Each route owns its query shape; the @ menu and the LLM tool stop fighting over a unified codepath.

3. Query: a single nested multi_match bool_prefix operator: and against discovery_labels.value and its auto-generated _2gram / _3gram shingle subfields. All-but-last analyzed tokens must match indexed terms exactly; only the trailing partial behaves as a prefix. Per-nested-entry operator: and enforcement means typed tokens must all land on the same entry.

4. Response shape (live-tested) — indexed record { type: 'connector', title: 'GitHub Connector' } with a producer-supplied [{ value: 'github', kind: 'tagline' }]. After the indexer auto-prepends title and type, the record's discovery_labels hold three entries: title GitHub Connector, type connector, tagline github. The route returns the entries that satisfied the query as matched_discovery_labels in ES score order.

   POST /sml/_autocomplete with body { "query": "git" } — "git" is a prefix (last/only token), so it prefix-matches all entries whose tokens start with git. Title and tagline contain such a token; type connector does not.

   {
     "total": 1,
     "results": [{
       "id": "connector:gh-1:abc",
       "type": "connector",
       "title": "GitHub Connector",
       "origin_id": "gh-1",
       "matched_discovery_labels": [
         { "value": "github",           "kind": "tagline" },
         { "value": "GitHub Connector", "kind": "title"   }
       ]
     }]
   }

   POST /sml/_autocomplete with body { "query": "github c" } — "github" is now a confirmed (non-trailing) token and must match an indexed term exactly; "c" is the trailing partial and may match as a prefix. Only the title entry has both: an exact github token and a c-prefix token (Connector). The tagline lacks c-prefix; the type lacks github.

   {
     "total": 1,
     "results": [{
       "id": "connector:gh-1:abc",
       "type": "connector",
       "title": "GitHub Connector",
       "origin_id": "gh-1",
       "matched_discovery_labels": [
         { "value": "GitHub Connector", "kind": "title" }
       ]
     }]
   }

   matched_discovery_labels ships as a server capability for the UI to lean on when ready. Today's @ menu doesn't render it yet — rows still display as ${type}/${title} with client-side EuiHighlight bolding the typed query (the existing UX), and the new field is available for future iteration into kind-aware badges. highlighted is intentionally absent on entries returned by this route — see "Known limitation: no highlights" below.

5. Bonus — kbn-storage-adapter: gains an optional analysis field on IndexStorageSettings, threaded through to the index template's settings.analysis. Added by the earlier commit on this branch for the edge_ngram pivot; retained as a generic capability so future consumers can declare custom analyzers without bypassing the adapter's bootstrap flow. Backwards-compatible. Ignored on serverless. Owners: @elastic/observability-ui, @elastic/kibana-core.

Known limitation: no highlights from this route (ES bug elastic#53744)

SAYT + bool_prefix + nested + inner_hits highlight is the exact configuration tracked in elastic/elasticsearch#53744 ("search_as_you_type, prefix queries and broken highlighting"), open since 2020-03 and explicitly low-priority per the ES Search team. Root cause documented in elastic/elasticsearch#49069: SAYT's _index_prefix subfield indexes position-spanning ngrams with offsets that span from the prefix start to the end of the entire field — the highlighter can't wrap per-word.

We keep the highlight config in the request (forward-compatible for when the bug is fixed) but ES returns no snippet, so matched_discovery_labels entries arrive without highlighted. The @ menu UI continues to bold the typed query client-side via EuiHighlight on the displayed type/title — that part is unchanged from the pre-PR UX, so the user always sees which substring matched, with or without server-side highlights.

Why we accept this rather than work around it

• prefix query directly: rewritten internally to a bool of term queries over every indexed term sharing the prefix. Expensive at query time; the whole point of SAYT/edge_ngram is to move that work to index time.

• Custom edge_ngram analyzer (commit 1 of this branch): highlights cleanly but produces loose matches across confirmed tokens. Worse typeahead UX than no highlight.

• Hybrid AND (live-tested but not taken):

  bool { must: [match(edge_ngram), match_bool_prefix(plain text)] }
  

  Tight matching from the match_bool_prefix clause + working highlights from the edge_ngram clause (the highlighter doesn't break on a match against an edge_ngram-analyzed plain text field). Costs an extra subfield on discovery_labels.value (with index_options: freqs, norms: false it's marginal — <5% of the discovery_labels postings size). Adds dual scoring. Mid-typing UX gap remains: "git c" returns 0 because "git" isn't a complete indexed token.

  Slotted for PR3 if real users feel the missing highlights — at that point we'd also have data on whether the mid-typing gap is a real problem.

For PR2 we ship the simpler, tighter SAYT path and accept no highlights as a documented limitation.

Schema

discovery_labels: nested { value, kind }

• value is search_as_you_type. ES auto-generates _2gram, _3gram, and _index_prefix subfields used by bool_prefix.
• The indexer auto-prepends {value: chunk.title, kind: 'title'} and {value: chunk.type, kind: 'type'} to every record, plus any entries the producer provides (taglines, nicknames, categories, etc.). Type writers don't change their SmlChunk shape.

Drops title_autocomplete (top-level SAYT field) and the .autocomplete subfield from type. Title's and type's autocomplete reach now goes through their auto-prepended discovery_labels entries; type stays as plain keyword for exact filtering.

Route + service

POST /internal/agent_context_layer/sml/_autocomplete — dedicated route for the @ menu, separate from /sml/_search.

buildSmlAutocompleteQuery: a single nested multi_match bool_prefix operator: and against discovery_labels.value and its _2gram / _3gram subfields. inner_hits returns the matched entries with their kind. operator: and is evaluated per nested entry, so an entry that contains only a subset of the typed tokens is excluded — see the "github c" example above where neither the type entry nor the tagline entry is returned (the type lacks a github-token; the tagline lacks a c-prefix-token).

Response: { id, type, title, origin_id, matched_discovery_labels? }. No content, no description, no score — order is sufficient. The @ menu today renders ${type}/${title} with EuiHighlight client-side bolding (same as pre-PR); matched_discovery_labels is server-side capability ready for FE engineers to adopt when they want kind-aware row badges.

The route accepts the same filters?: SmlSearchFilters body parameter as /sml/_search — same shape, same buildTypeFilters helper, same per-type allow-list semantics. This is what restores the agent-centric connector scoping (#267333) for the @ menu: an agent with configuration.connector_ids: [...] sees only the listed connectors in the typeahead. Other SML types are unaffected.

filterResultsByPermissions is generic over the result type and shared with the search path.

kbn-storage-adapter change

Adds an optional analysis field to IndexStorageSettings, threaded through to the index template's settings.analysis block. Introduced in commit 1 for the edge_ngram pivot; SML no longer uses it after commit 2's switch to SAYT, but the storage-adapter capability is retained as it's broadly useful for future consumers that need to declare custom analyzers without bypassing the adapter's bootstrap flow. Backwards-compatible — field is optional. Ignored on serverless ES (no index-level settings).

Owners: @elastic/observability-ui, @elastic/kibana-core.

Test plan

• node scripts/eslint --fix on touched files.
• node scripts/type_check --project x-pack/platform/plugins/shared/agent_context_layer/tsconfig.json.
• node scripts/jest x-pack/platform/plugins/shared/agent_context_layer/server/services/sml/ — all pass.
• node scripts/jest src/platform/packages/shared/kbn-storage-adapter/src/index_adapter/ — all pass.
• node scripts/jest x-pack/platform/plugins/shared/agent_builder/public/application/components/conversations/conversation_input/message_editor/command_menu/menus/sml/ — sml.test.tsx + use_prefetch_sml.test.tsx cover the autocomplete + filter wiring.
• node scripts/jest x-pack/platform/plugins/shared/agent_builder/public/application/hooks/sml/use_sml_autocomplete.test.tsx — new hook test, including the filters arg.
• FTR (real-ES) agent_builder_api_integration SML autocomplete tests exercise the route end-to-end against indexed discovery_labels entries; assert matched_discovery_labels and kind. highlighted is asserted absent (documents the WIP: Move agg_types into data shim plugin elastic/kibana#53744 known limitation).
• Manual: open the @ menu in an agent conversation, confirm typeahead. Update the agent to a subset of connector_ids, re-open the menu — only the listed connectors appear.